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Introduction 



Since the piloting of value-added progress measures by the Ohio Department of Education 
in 2007, we at the Thomas B. Fordham Institute have repeatedly been asked by business 
people, journalists, philanthropists, educators, and others to explain "value-added" and 
what it means for schools and children. In order to help answer this question and 
others— such as "How is it used?" "Should it replace standards-based assessments?" and 
"Is it a fairer measurer of school performance?"— we volunteered to produce a short 
primer to help non-specialists understand the basics of "value-added." 

This stuff can be complicated. Indeed, "value-added" analysis can be carried out in 
various ways and with different emphases. These approaches are much discussed and 
argued over by experts. Each has pros and cons, relies on certain assumptions, and is 
fraught with some uncertainties. Let us be clear, too, that in seeking to explain the 
approach that Ohio has embraced, we don't necessarily endorse it. But it's important to 
know how it works and can best be used for educational improvement. 

In striving to simplify something that is inherently complex, one always runs the risk of 
over-simplifying or not getting something quite right. We have done our best to avoid 
those perils without miring the reader in gobbledy-gook. In this effort, we are indebted to 
many people. 

First and foremost, we commend state lawmakers, the State Board of Education, the Ohio 
Department of Education, and Battelle for Kids for their leadership over the past five years 
in developing and implementing Ohio's value-added analysis system. Ohio is one of the 
first states in the country to measure and report on both student achievement (do 
students know the expected course content?) and student progress (how much have they 
grown in skills and knowledge over the course of the year?). As the result of this effort, 
the state has developed a richer and more sophisticated way to track student and school 
performance. These data empower school leaders and teachers to target more accurately 
instruction and programs to meet the needs of their pupils while providing parents, 
taxpayers, voters, and policymakers with a fairer and better-informed way to hold schools 
and districts accountable for student performance. 

In the effort to translate the complexities of value-added analysis into user-friendly 
language, we reached out to two people who are uncommonly adept at describing and 
explaining complex education issues: Bryan Hassel and Jacob Rosch of the policy and 
research organization Public Impact. 

We also owe much gratitude to the reviewers who helped refine and improve this 
document: Mary Peters, Battelle for Kids; Kristina Phillips-Schwartz, Cincinnati Business 
Committee; Ann Bischoff and Mark Real, KidsOhio.org; and Jeremy Shorr, Mentor City 
Schools. 

At the Fordham Institute, we thank Mike Petrilli and Amber Winkler for their invaluable 
feedback. We also extend thanks to former Fordham Institute staffer Quentin Suffren for 
his editing skills and to Jayson Kinsey of Transcend Media for his design work. 

Our goal at Fordham is to improve the educational opportunities, options, and 
performance of this state's students, particularly the neediest among them. Toward that 
end we dedicate our work and this primer. 

Chester E. Finn > Jr., President 

Terry Ryan, Vice President for Ohio Programs and Policy 
Emmy L. Partin, Researcher 
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Executive Summary 

With the passage of Senate Bill 1 in 2001 and in compliance with the federal No Child Left 
Behind act, Ohio created a standards-based academic accountability system based on 
student achievement on statewide assessments in grades 3 -8 as well as a high school 
graduation test. In 2007, Ohio piloted a "value-added" accountability component that 
measures student academic progress— how much students have learned over time (see 
more on page 3). Beginning in August 2008, value-added is fully integrated into Ohio's 
academic accountability system. Its purpose is to show whether and how much knowledge 
and skills schools and districts are adding to their students' learning over the course of 
one or more school years (see more on page 4). 

Ohio's new value-added analysis uses complex calculations (see more on page 5) to 
report school-wide and district-wide pupil-achievement growth in three categories: Above 
Expected Growth (+), Expected Growth (^), or Below Expected Growth (-). 

- Above Expected Growth (+) indicates that the students in a school or district 
made greater than expected progress. These schools and districts are "adding 
value." 

■ Met Expected Growth (J) indicates that students made the amount of expected 
progress in one school year. Districts and schools in this category are still adding 
value, but not as much as those schools rated as Above Expected Growth (+). 

Below Expected Growth (-) indicates that students made less progress than 
the state expected. 

Value-added analysis can identify where students are markedly improving or consistently 
struggling and can suggest how well students in those schools or districts are likely to 
perform in the future. This information can help policymakers and system leaders identify 
successful schools and districts and intervene when progress lags. It can also help 
educators adjust curriculum and instruction and design more effective strategies to help 
individual students (see more on page 8). 

While value-added analysis is a powerful tool that policymakers and the public can use to 
judge which schools and districts are making sufficient academic progress, it does not tell 
the whole story of academic performance. Progress measures alone cannot reveal what 
students actually know or whether schools and districts have reached the academic goals 
set for them. To understand the performance of schools and districts more fully, 
policymakers (and education practitioners, parents, community leaders, et al.) should 
consider both how much progress students are making (value-added) and what 
proportion of those students have mastered the state's academic content standards (see 
more on page 7). 
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Two Ways to Measure Learning: 

Achievement and Progress 

Beginning in August 2008, Ohio's academic accountability system is reporting a new 
component— "value-added analysis"— that measures student academic growth over time. 
Value-added changes how Ohio evaluates its public schools. This primer is designed to 
help readers better understand how value-added is calculated, what it can (and cannot) 
reveal, and how Ohioans can use value-added information to improve student learning 
outcomes. 

We can assess how much students are learning in two broad ways: by measuring what 
students know and can do at a particular point in time (their "achievement" or 
"proficiency"), and by measuring how much students have improved (their "progress" or 
"growth") over a period of time. 

A measure of achievement gives us a snapshot of what students know at the time they 
were tested. A measure of achievement could, for example, report whether a student was 
reading at grade level or what proportion of 5th grade students in a school were reading 
at grade level. A measure of progress, by contrast, tells us how much a student has 
learned over a designated period of time. A measure of progress might report that a 
student's reading score improved 100 points from 4th grade to 5th grade, or that most 
students in a school's kindergarten began the year far behind grade level but ended the 
year above grade level. Policymakers, educators, and the public can use achievement and 
progress measures differently to gauge how schools, districts, or the state as a whole are 
performing. One might find, for example, that a school has many high-achieving pupils 
enrolled in it but that they are not making very much progress while enrolled there. 
Conversely, a school may be helping its students to make great progress though their 
overall achievement is still not what it should be. 

One advantage of achievement measures is that they hold all students and schools to the 
same standard of proficiency: they can tell us whether students or schools have reached 
goals set for them by, in this case, the state's academic standards in particular subjects 
and grades. Consider, for example, a student who enters kindergarten far behind the 
state's standards. Even if she makes a year's worth of progress each year during 
elementary school, she will still be far behind when she leaves 8th grade. If we only held 
schools accountable for progress, we would deemphasize the importance of determining 
what the student knows and can do— and of whether this particular student is truly 
prepared to succeed in high school and beyond. 

Yet examining progress is also important. Consider the following example, which follows 
the achievement of Student A (red line) and Student B (green line): 

In 6th grade, both students are 
at grade level. If we only look at 
achievement at that point, the 
students appear to be identical. But 
Student B has improved dramatically 
over two years, while Student A's 
achievement has actually declined 
luring that period. 



Adapted from Battelle for Kids 




Thomas B. Fordham Institute 



3 



Looking only at student achievement can also be misleading. Consider two schools, each 
with 75 percent of their students at grade level. One school, however, rose to the 75 
percent marker from 47 percent two years earlier. The other's performance, by contrast, 
dropped from 82 to 75 percent during that same time. While a snapshot of the two schools 
using measures of achievement shows the same picture at one point in time, their 
performance trajectories are quite different. 

Progress, if measured effectively, can tell us where one cohort of students is improving or 
consistently struggling and how well students in that school or district are likely to 
perform in the future. This way, curriculum can be appropriately adjusted and more 
effective academic interventions can take place. In short, we need both achievement and 
progress measures to make sense of student and school performance and to determine 
what to do about it. 



What is Value-Added? 

One challenge of measuring progress is that it can be quite difficult to judge the 
significance or magnitude of a student's gains or those of a group of students. Imagine, 
for example, that a student scores 75 points in one year and 82 the next. We can see that 
she made progress (or "a gain") of 7 points. But we don't know whether her gain was 
large, small, or in-between. In this example, if most children only progress by one or two 
points each year, the student's 7-point gain looks large. But it looks small if students 
typically leap ahead by 10 or 20 points during a year. 

In other words, if we don't know what to expect, we can't judge the significance of a 
student's progress. (For more, see the box Is Progress Significant?) 

Value-added is one way to make that kind of judgment. Value-added methods are tools 
that analysts have devised to help us understand the gains that students are or are not 
making while in school. This can be done in many ways, and different analysts and states 
have developed various approaches. 

In Ohio, the goal was to determine whether students' gains are large, small, or 
in-between by comparing them to the gains we would expect similar students to make. If 
students in a school made better-than-expected progress, under the Ohio system we may 
conclude that the school "added value" to the students' learning (i.e., it helped the 
students learn more than was expected). 



Achievement 



Progress 



Compares students’ performance to 
a standard. 



Measures a student’s performance 
at a single point in time. 



Relates to a student’s family 
background 




Critical to ensuring a student's future 



A more complete picture 
of student learning. 




Critical to a student’s post 
secondary opportunities 



academic success. 
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Is Progress Significant? 

Figures la and lb show the progress a hypothetical student made from 2006 to 2007. In Figure la 
we see only the student's progress, in Figure lb we see that progress compared with the progress 
an average student makes (called "expected" progress in Ohio). Was this student's progress large, 
small, or in-between? 



84 

82 

80 

78 

76 

74 

72 

70 



Figure la; One Slu deni's Growth 



2007 


K 






1 








J 








—JftJ 


1 



140 

120 

100 

80 

60 

40 

20 

0 



Figure Itr Student Growth vs. Average Student Growth 



Student's Tesi Scores 



2006 2007 



i r~r 




2007 



2006 IT 




Student's Test Scores 



Average Test Scores 



From 2006 to 2007, this student's score rose from 75 to 82, a gain of 7 points. When viewed 
individually, this student's progress appears large; however, when compared to the progress an 
average student makes (from 75 to 100, or a gain of 25 points) this gain appears small. 



How Does Ohio Calculate Value-Added? 

Ohio's measurement of value-added is based on the SAS Institute's Education 
Value-Added Assessment System (EVAAS) (http://www.sas.com/evaas), also known as 
the "Sanders model" after its inventor, William Sanders. Versions of this model are 
currently being used in Colorado, Iowa, New Hampshire, North Carolina, Pennsylvania, 
and Tennessee. EVAAS is a powerful but complicated way to determine whether students' 
academic progress is meeting or exceeding expectations. 

To understand how it works, think about comparing anticipated and actual scores. Each 
year, students across the state take the Ohio Achievement Tests (OATs). The performance 
of individual students and cohorts of students one year can be used to predict their 
performance in the next year. 



For instance, imagine that we know from 
experience that most students who score 85 
points on the OAT for math in the 3rd grade will 
score 91 points in the 4th grade, a gain of 6 
points. 

If a student scored 85 points in the 3rd grade 
and then scored 128 points in the 4th grade, a 
gain of 43 points, we would say that this 
student has made significant progress by 
growing far more than expected. For more see 
the box Actual vs. Expected Performance. 

For Ohio, EVAAS conducts a similar, though 
more complicated, analysis. Instead of just 
comparing a student's gain to the average gain, 

EVAAS develops a customized prediction of each student's progress based, if possible, on 
the student's own academic record as well as that of other students over multiple years, 
with statewide test performance serving as an anchor. 



The Ohio Achievement Tests (OATs) 

The Ohio Achievement Tests or OATs are 
given to all students in Ohio in grades 3-8 
in reading and math, grades 5 and 8 in 
science and social studies, and grades 4 
and 7 in writing. 

The OATs assess whether students have 
mastered grade-level content in each 
subject. High achievement on the OATs is 
correlated to a deep understanding of the 
skills and knowledge set forth in Ohio's 
grade-level academic standards. 

At this time, value-added analysis is only 
used by Ohio to gauge student progress in 
reading and math in grades 4-8. 
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EVAAS calculates a number for each student in reading and math called a "value-added 
gain." A value-added gain is the difference between what a student scored in a given 
subject and what EVAAS predicted that student would score. If a student exceeds these 
expectations, she will have a positive value-added gain. If she falls short of expectations, 
her value-added gain will be negative. And if she meets expectations, her value-added 
gain will be at or near 0. (For more, see the F.A.Q. section at the end.) 



Actual vs. Expected Performance 

Figure 2 shows a simulation of what a value-added analysis might look like. Each student receives 
a predicted score based on the student's own academic record and that of other students over 
multiple years. This "expected score" is then compared with the student's actual score in a given 
year. Student A made greater than expected growth because he scored higher than his expected 
score. Students B and C made expected growth because they scored at or near their expected 
scores, while Student D made less than expected growth because he scored below his expected 
score. Overall, the four students, taken as a group (a "classroom", say) made greater than expected 
growth. 



Figure 2: Expected vs. Actual Performance U 2007 Score ■ Expected 2008 D Actual 2008 




Student A Student B Student C 



Student D 



Overall 



Reporting Value-Added 

By compiling the value-added gain of every student in a school, it is possible to determine 
whether that school as a whole made progress that was above, at, or below expectations. 
In the same way, by compiling the performance of every student in every school in a 
district, it is possible to determine whether the district as a whole made progress that was 
above, at, or below expectations 

Report Cards 

Ohio's school and district report cards will report school-wide and district-wide growth in 
three categories: 

• Above Expected Growth (+) • Met Expected Growth (V) 

• Below Expected Growth (-) 

Each school and district will receive a rating in reading and math for grades 4 through 8, 
as well as a composite rating. 

A rating or Above Expected Growth (+) indicates that the students in a school or district, 
taken as a group, made greater than expected progress. It is possible to assume a school 
or district that is continually rated Above Expected Growth (+) is having a significant 
impact on its students' achievement, This school or district is "adding value" to its 
students' educations. It should be noted, however, that school-wide and district-wide 
value-added results may conceal considerable variation by student and demographic 
group within that school or district. 

Met Expected Growth («/) indicates that students made the amount of expected progress 
in one school year. Districts and schools in this category arc still adding value, but not as 
much as those schools rated Above Expected Growth (+). 

A rating of Below Expected Growth (-) means that students made less progress than they 
should have. While there are many possible reasons for this, a school or district that is 
repeatedly rated as Below Expected Growth (-) is clearly not accomplishing as much for 
its pupils' learning as the state expects. 
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Value-Added and Ohio's Report Card Designations 

Ohio currently uses a ratings-based accountability system to track school performance. 
Based on a set of standard criteria, schools and districts receive a designation that rates 
their performance. Through 2007, five designations were possible: Excellent, Effective, 
Continuous Improvement, Academic Watch, and Academic Emergency. 

In the past, four criteria were used to determine a school's rating: 

• State-specific Indicators, such as percent of students proficient in various 
subjects and grades, graduation rates, and attendance rates; 

a Adequate Yearly Progress (AYP) status, the federally-mandated determination of 
whether enough students in each of a school's subgroups are meeting standards; 

a Performance Index (PI) scores, which combine the results of all statewide tests 
into a single scale that runs from 0 to 120; and, 

* Growth in Performance Index, to give schools credit for progress over time. 



Beginning in 2008, the Growth in Performance Index is replaced by value-added as the 
state's measure of progress. Schools' ratings will be calculated based on their 
performance on the first three criteria, and then value-added results will be factored in. 
Schools that show Above Expected Growth for two years in a row will move up one rating. 
Any school that shows Below Expected Growth for three years in a row will move down 
one rating. Additionally, a new designation, "Excellent with Distinction," will be added to 
the ratings. This designation is only possible for schools rated Excellent that also show two 
/ears of Above Expected Growth. 



Power User Reports 

"Power User" reports also allow Ohioans to obtain 
more in-depth value-added data about schools and 
districts. These reports can be generated online at 
the Ohio Department of Education's website. While 
report cards will only show schools' and districts' 
value-added classifications (+V, or -), Power User 
reports show each school's and district's specific 
value-added gain, value-added classification, and 
standard error for reading and math, overall, and 
within grade levels. 

Value-Added Gain is a number that tells us how a 
school's or district's students performed in relation 
to the state's expectations. The larger the value, 
the larger the difference between the students' test 
scores and their expected test scores. A score of 0 
would mean that students exactly met 
expectations. 

Value-Added Classification explains the significance 
of a value-added gain. Note that a school or district 
may have a positive value-added gain but the 
value-added result may be classified as just 
meeting, rather than exceeding, expected gain. 
This happens when the value-added gain falls 
within the Value-Added Standard Error. (For more, 
;ee the box Understanding Standard Error.) 



Understanding 

Standard Error 



A student's expected score is actually an 
estimate. EVAAS predicts lhal a sludenl's 
result will fall within o range, the 
prediction pius-or-mlnus some number. 
That plus-or-minus number is called the 
standard error" Standard error is 
Influenced by factors like sample size and 
haw many years of data are available. 

For etfampte, Ohio might estimate that a 
student should gatn ) 0 points . But 
because this Is on estimate, Ohio also 
calculates the standard error to be 2.0. 
AS a re soli, it predicts Shal She sludeat will 
gain between 6 and 1 2 points: tG - 2 - 6. 
and TO + 2 = 12. For a student's gain to 
be deemed above or below expectations, 
H must fall DuSsJde this range 
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Using Value-Added to Improve Ohio's Schools 

With the addition of value-added analysis, Ohio has a new and powerful way to assess the 
effectiveness of its schools. With value-added data, policymakers, educators, parents, and 
community leaders are able to judge which schools and districts are helping students 
make significant academic progress. 

Even though progress is important, it is crucial to remember that value-added is only one 
piece of the assessment puzzle. It does not reveal how much students actually know or 
can do. A value-added score can tell us how much progress students are making, but 
progress alone cannot reveal whether students are prepared for success in high school 
and beyond. 

To assess the performance of a school or district, it is important to examine both progress 
and achievement. Any school or district in Ohio can be placed on the chart below based 
on its Performance Index (which measures achievement) and its value-added 
classification (which measures progress). A school with high achievement (a Performance 
Index score greater than 100) but low progress (a value-added rating of Below Expected 
Growth) would be placed in the top left cell. A school with low achievement (a 
Performance Index score less than 80) but high progress (a value-added rating of Above 
Expected Growth) would be placed in the bottom right cell. While it is possible for schools 
to fall into any category, ideally a school or district would have high levels of achievement 
and high levels of progress— represented by the upper right cell of the figure below. If 
used in conjunction with achievement measures, Ohio's new value-added analysis can 
help the state move more and more schools and districts in that direction. 
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Frequently Asked Questions 

How can teachers, schools, and districts use value-added to improve schools? 

Value-added gives educators unprecedented information about individual students' 
growth. With value-added information, it is possible to target teaching, curriculum, or 
academic interventions to specific areas where student growth is lagging. Furthermore, by 
giving educators valuable information about the progress that each individual student 
should make, from wherever he or she started, we can help focus on what really 
matters— student learning. 

Can I see the value-added score for a particular teacher or classroom? 

Your superintendent can dispense that information at his or her discretion. 

Why does value-added use expected scores? Wouldn't it be easier just to look at 
the change in a student's scores from one year to the next? 

Because students are expected to learn new material each year, their score should 
naturally increase each year. How much it should increase is what the expected score 
calculates. Value-added analysis looks at whether students progressed by more or less 
than expected. For more, see the box Is Progress Significant?, page 5. 

How does EVAAS predict what a student will score? 

EVAAS does not actually predict a student's raw score; instead EVAAS conducts an 
analysis called value-added mean gain. A student's past scores are converted to a 
percentile-like index score, called a normal curve equivalent (NCE). This describes where 
a student's scores fall in relation to his or her peers. A student is expected to at least 
maintain his or her position on the NCE scale from year to year. Imagine a student has 
an NCE of 42 in math one year and an NCE of 42 the next year. This student "met 
expectations" and her value-added mean gain will be 0. If the student's NCE of 42 in math 
fell to 35 the following year, her value-added mean gain would be -7. 

Does value-added take into account students' socioeconomic status (race, 
ethnicity, or income)? 

Because socioeconomic factors like race, ethnicity, and income should affect a student 
similarly from year to year, these factors do not have to be accounted for separately. 
Controlling for a student's past academic history should account for any socioeconomic 
effect. In other words, in the EVAAS model each child serves as his or her own control. 
This reduces the need to control for socioeconomic factors such as race, ethnicity, and 
income. For more information on why EVAAS does not explicitly consider demographic 
factors, see: http://www.sas.com/evaas. 

If a value-added gain is negative or below expected, does that mean students' 
scores dropped? 

Not necessarily. If a student's score increases by less than expected, her value-added gain 
will be negative. Even though her score improved, it did not improve enough to meet 
expectations. For more see the box Actual vs. Expected Performance, page 6. 

If students, schools, or districts earned a rating of Below Expected Growth, does 
that mean they got worse? 

Not necessarily. Below Expected Growth only means that students did not progress as 
much as expected. If students do not make expected growth, it is more accurate to say 
that students' did not meet their potential or did not accomplish what the state believes 
they should. 
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F.A.Q. Continued 



How can a value-added gain be negative or positive, but still be coded Met 
Expected Growth? Shouldn't only value-added scores of 0 be considered 
expected growth? 

If any value-added gain is within the standard error, then it is too small to be considered 
above or below expected growth. For example, with a standard error of 1, gains of both 
+ .8 and -.6 would be considered expected growth, since they are within +1 and -1 units 
of 0. For more on this point, see the box Understanding Standard Error, page 7. 

What if a student has changed schools or districts? How will his or her 
value-added be calculated? 

Schools and districts are only accountable for students enrolled for a full academic year. 
If a student transfers to a new school within the same district, that student's value-added 
history will "follow" him or her. A student with missing information can still receive a 
value-added score as long as the child has adequate previous test data. For more 
information on how the EVAAS system calculates value-added when some student data is 
missing, see: http://www.sas.com/evaas. 

Where can I get more information about value-added? 

The Ohio Department of Education's website contains additional information about Ohio's 
accountability system, including value-added. From the department's homepage 
(http://www.ode.state.oh.us) type "value-added" into the search box. For more 
information about the SAS EVAAS model see: http://www.sas.com/evaas. 

You may also contact your local school for information about value-added and how the 
school uses it to improve student learning outcomes. 

How is Battelle for Kids' Project SOAR different from Ohio's new value-added 
analysis? 

Project SOAR uses different methodology to compute value-added gains. Under the SOAR 
system, growth is measured in relation to the average student growth in one year. This 
creates a distribution where approximately half of the gains will be above expected, and 
half of the gains will be below expected. Under Ohio's system, the distribution of results 
from 2007 acts as an anchor from which to measure growth. How well students did in 
2007 is measured against how well they did in 2006. This means growth may not fall into 
an even distribution: more than 50 percent of students can make above expected growth 
or more than 50 percent can make below expected growth. 

For more information see the Ohio Department of Education's accountability website: 
http ://www. ode. state, oh. us/GD/Templates/Pages/ODE/ODEPrimary.aspx?page=2&Topic 
RelationID=1065. 

For information about Project SOAR, see Battelle for Kids' website: 
http://www.battelleforkids.org. 

How can I access school or district value-added scores? 

From the Ohio Department of Education's homepage (http://www.ode.state.oh.us), enter 
"Data Warehouse" into the search box. After accessing the Data Warehouse, you may find 
value-added scores for schools and districts in the "Ratings" folder. 

Will high schools receive value-added results? 

No. Multiple years of data are needed to produce value-added results and Ohio high school 
students are currently assessed only in the 10th grade. 
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